
<!doctype html>
<html lang="en" class="no-js">
  <head>
    
      <meta charset="utf-8">
      <meta name="viewport" content="width=device-width,initial-scale=1">
      
      
      
      <link rel="icon" href="../../../../static/images/favicon.png">
      <meta name="generator" content="mkdocs-1.3.0, mkdocs-material-8.2.8">
    
    
      
        <title>Spark 生产集群性能调优之路 - WL4G DOCS</title>
      
    
    
      <link rel="stylesheet" href="../../../../assets/stylesheets/main.644de097.min.css">
      
        
        <link rel="stylesheet" href="../../../../assets/stylesheets/palette.e6a45f82.min.css">
        
      
    
    
    
      
        
        
        <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
        <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
        <style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
      
    
    
      <link rel="stylesheet" href="../../../../static/css/util.css">
    
    <script>__md_scope=new URL("../../../..",location),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
    
      

    
    
  </head>
  
  
    
    
      
    
    
    
    
    <body dir="ltr" data-md-color-scheme="default" data-md-color-primary="" data-md-color-accent="">
  
    
    
      <script>var palette=__md_get("__palette");if(palette&&"object"==typeof palette.color)for(var key of Object.keys(palette.color))document.body.setAttribute("data-md-color-"+key,palette.color[key])</script>
    
    <input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
    <input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
    <label class="md-overlay" for="__drawer"></label>
    <div data-md-component="skip">
      
        
        <a href="#spark" class="md-skip">
          Skip to content
        </a>
      
    </div>
    <div data-md-component="announce">
      
    </div>
    
      <div data-md-component="outdated" hidden>
        <aside class="md-banner md-banner--warning">
          
        </aside>
      </div>
    
    
      

<header class="md-header" data-md-component="header">
  <nav class="md-header__inner md-grid" aria-label="Header">
    <a href="../../../.." title="WL4G DOCS" class="md-header__button md-logo" aria-label="WL4G DOCS" data-md-component="logo">
      
  <img src="../../../../static/images/mylogo.jpeg" alt="logo">

    </a>
    <label class="md-header__button md-icon" for="__drawer">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2z"/></svg>
    </label>
    <div class="md-header__title" data-md-component="header-title">
      <div class="md-header__ellipsis">
        <div class="md-header__topic">
          <span class="md-ellipsis">
            WL4G DOCS
          </span>
        </div>
        <div class="md-header__topic" data-md-component="header-topic">
          <span class="md-ellipsis">
            
              Spark 生产集群性能调优之路
            
          </span>
        </div>
      </div>
    </div>
    
      <form class="md-header__option" data-md-component="palette">
        
          
          
          <input class="md-option" data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme="default" data-md-color-primary="" data-md-color-accent=""  aria-label="Switch to dark mode"  type="radio" name="__palette" id="__palette_1">
          
            <label class="md-header__button md-icon" title="Switch to dark mode" for="__palette_2" hidden>
              <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M17 6H7c-3.31 0-6 2.69-6 6s2.69 6 6 6h10c3.31 0 6-2.69 6-6s-2.69-6-6-6zm0 10H7c-2.21 0-4-1.79-4-4s1.79-4 4-4h10c2.21 0 4 1.79 4 4s-1.79 4-4 4zM7 9c-1.66 0-3 1.34-3 3s1.34 3 3 3 3-1.34 3-3-1.34-3-3-3z"/></svg>
            </label>
          
        
          
          
          <input class="md-option" data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme="slate" data-md-color-primary="" data-md-color-accent=""  aria-label="Switch to light mode"  type="radio" name="__palette" id="__palette_2">
          
            <label class="md-header__button md-icon" title="Switch to light mode" for="__palette_1" hidden>
              <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M17 7H7a5 5 0 0 0-5 5 5 5 0 0 0 5 5h10a5 5 0 0 0 5-5 5 5 0 0 0-5-5m0 8a3 3 0 0 1-3-3 3 3 0 0 1 3-3 3 3 0 0 1 3 3 3 3 0 0 1-3 3z"/></svg>
            </label>
          
        
      </form>
    
    
      <div class="md-header__option">
        <div class="md-select">
          
          <button class="md-header__button md-icon" aria-label="Select language">
            <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m12.87 15.07-2.54-2.51.03-.03A17.52 17.52 0 0 0 14.07 6H17V4h-7V2H8v2H1v2h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04M18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12m-2.62 7 1.62-4.33L19.12 17h-3.24z"/></svg>
          </button>
          <div class="md-select__inner">
            <ul class="md-select__list">
              
                <li class="md-select__item">
                  <a href="/en/" hreflang="en" class="md-select__link">
                    English
                  </a>
                </li>
                
                <li class="md-select__item">
                  <a href="/zh/" hreflang="zh" class="md-select__link">
                    简体中文
                  </a>
                </li>
                
            </ul>
          </div>
        </div>
      </div>
    
    
      <label class="md-header__button md-icon" for="__search">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5z"/></svg>
      </label>
      <div class="md-search" data-md-component="search" role="dialog">
  <label class="md-search__overlay" for="__search"></label>
  <div class="md-search__inner" role="search">
    <form class="md-search__form" name="search">
      <input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
      <label class="md-search__icon md-icon" for="__search">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5z"/></svg>
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12z"/></svg>
      </label>
      <nav class="md-search__options" aria-label="Search">
        
        <button type="reset" class="md-search__icon md-icon" aria-label="Clear" tabindex="-1">
          <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12 19 6.41z"/></svg>
        </button>
      </nav>
      
        <div class="md-search__suggest" data-md-component="search-suggest"></div>
      
    </form>
    <div class="md-search__output">
      <div class="md-search__scrollwrap" data-md-scrollfix>
        <div class="md-search-result" data-md-component="search-result">
          <div class="md-search-result__meta">
            Initializing search
          </div>
          <ol class="md-search-result__list"></ol>
        </div>
      </div>
    </div>
  </div>
</div>
    
    
  </nav>
  
</header>
    
    <div class="md-container" data-md-component="container">
      
      
        
          
            
<nav class="md-tabs" aria-label="Tabs" data-md-component="tabs">
  <div class="md-tabs__inner md-grid">
    <ul class="md-tabs__list">
      
        
  
  


  
  
  
    <li class="md-tabs__item">
      <a href="../../../.." class="md-tabs__link">
        Getting Started
      </a>
    </li>
  

      
    </ul>
  </div>
</nav>
          
        
      
      <main class="md-main" data-md-component="main">
        <div class="md-main__inner md-grid">
          
            
              
              <div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
                <div class="md-sidebar__scrollwrap">
                  <div class="md-sidebar__inner">
                    

  


  

<nav class="md-nav md-nav--primary md-nav--lifted md-nav--integrated" aria-label="Navigation" data-md-level="0">
  <label class="md-nav__title" for="__drawer">
    <a href="../../../.." title="WL4G DOCS" class="md-nav__button md-logo" aria-label="WL4G DOCS" data-md-component="logo">
      
  <img src="../../../../static/images/mylogo.jpeg" alt="logo">

    </a>
    WL4G DOCS
  </label>
  
  <ul class="md-nav__list" data-md-scrollfix>
    
      
      
      

  
  
  
    
    <li class="md-nav__item md-nav__item--nested">
      
      
        <input class="md-nav__toggle md-toggle" data-md-toggle="__nav_1" data-md-state="indeterminate" type="checkbox" id="__nav_1" checked>
      
      
      
      
        <label class="md-nav__link" for="__nav_1">
          Getting Started
          <span class="md-nav__icon md-icon"></span>
        </label>
      
      <nav class="md-nav" aria-label="Getting Started" data-md-level="1">
        <label class="md-nav__title" for="__nav_1">
          <span class="md-nav__icon md-icon"></span>
          Getting Started
        </label>
        <ul class="md-nav__list" data-md-scrollfix>
          
            
              
  
  
  
    <li class="md-nav__item">
      <a href="../../../.." class="md-nav__link">
        Introduction
      </a>
    </li>
  

            
          
            
              
  
  
  
    <li class="md-nav__item">
      <a href="../../../../ABOUT_CN/" class="md-nav__link">
        About
      </a>
    </li>
  

            
          
        </ul>
      </nav>
    </li>
  

    
  </ul>
</nav>
                  </div>
                </div>
              </div>
            
            
          
          <div class="md-content" data-md-component="content">
            <article class="md-content__inner md-typeset">
              
                


<h1 id="spark">Spark 生产集群性能调优之路<a class="headerlink" href="#spark" title="Permanent link">&para;</a></h1>
<ul>
<li><a href="https://blogs.wl4g.com/archives/1248">关联参考: HBase 生产集群性能调优之路</a></li>
</ul>
<h2 id="1-spark-streaming">1. Spark Streaming 背压机制<a class="headerlink" href="#1-spark-streaming" title="Permanent link">&para;</a></h2>
<ul>
<li>推荐配置（默认 false）</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1"></a><a href="#__codelineno-0-1"><span class="linenos" data-linenos="1 "></span></a>--conf spark.streaming.backpressure.enabled<span class="o">=</span><span class="nb">true</span>
<a id="__codelineno-0-2" name="__codelineno-0-2"></a><a href="#__codelineno-0-2"><span class="linenos" data-linenos="2 "></span></a>--conf spark.streaming.kafka.maxRatePerPartition<span class="o">=</span><span class="m">8400</span> <span class="c1"># 用于控制吞吐量</span>
</code></pre></div>
<ul>
<li>1.1 如何估算设置？例需要 <code>10</code> 分钟的吞吐量控制在 <code>50,000,000</code>，Kafka 分区是 <code>10</code>，计算公式为:
<code>50,000,000 = maxRatePerPartition * kafkaPartitions * 10 * 60</code> (每秒)</li>
</ul>
<p>参考源码:
<a href="https://github.com/apache/spark/blob/v2.3.1/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala#L127">https://github.com/apache/spark/blob/v2.3.1/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala#L127</a></p>
<ul>
<li>1.2 场景：spark streaming kafka 程序首次启动的时候，一般会出现下面的情况。因为有数据挤压，所以会有很多在 queued 状态的 batch。
如果数据量特别大，可能会出问题，因此参数 <code>spark.streaming.kafka.maxRatePerPartition</code> 就尤为重要，他控制着 Spark Streaming 每秒每个分区最大的消费数量，使得挤压的数据可以慢慢消费。</li>
</ul>
<p><img alt="" src="https://blogs.wl4g.com:443/wp-content/uploads/2021/07/spark1.png" /></p>
<h2 id="2-spark">2. Spark 推测执行机制<a class="headerlink" href="#2-spark" title="Permanent link">&para;</a></h2>
<ul>
<li>
<p>解决颠簸流量导致数据倾斜，但会降低性能谨慎使用，默认 false；</p>
</li>
<li>
<p>推测任务是指对于一个Stage里面拖后腿的Task，会在其他节点的Executor上再次启动这个task，如果其中一个Task实例运行成功则将这个最先完成的Task的计算结果作为最终结果，同时会干掉其他Executor上运行的实例。spark推测式执行默认是关闭的，可通过spark.speculation属性来开启；</p>
</li>
<li>
<p>注: 亲经历在 spark 2.3.1 时开启推测后，偶尔性报错 <code>ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access</code>，暂时没有完全证实是这个原因？？？开启时注意一下</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1"></a><a href="#__codelineno-1-1"><span class="linenos" data-linenos="1 "></span></a>--conf spark.speculation<span class="o">=</span><span class="nb">true</span>
<a id="__codelineno-1-2" name="__codelineno-1-2"></a><a href="#__codelineno-1-2"><span class="linenos" data-linenos="2 "></span></a>--conf spark.speculation.interval<span class="o">=</span><span class="m">100</span>
<a id="__codelineno-1-3" name="__codelineno-1-3"></a><a href="#__codelineno-1-3"><span class="linenos" data-linenos="3 "></span></a>--conf spark.speculation.quantile<span class="o">=</span><span class="m">0</span>.9
<a id="__codelineno-1-4" name="__codelineno-1-4"></a><a href="#__codelineno-1-4"><span class="linenos" data-linenos="4 "></span></a>--conf spark.speculation.multiplier<span class="o">=</span><span class="m">1</span>.8
</code></pre></div>
<ul>
<li>2.1. 当spark.speculation设置为true时，就会对task开启推测执行，也就是在一个stage下跑的慢的tasks有机会重新启动；</li>
<li>2.2. spark.speculation.interval，100ms，Spark检测tasks推测机制的间隔时间；</li>
<li>2.3. spark.speculation.quantile，0.9，当一个stage下多少百分比的tasks运行完毕后才开启推测执行机制，0.9即90%的任务都运行完毕后开启推测执行；</li>
<li>2.4. spark.speculation.multiplier，1.5，一个task的运行时间是所有task的运行时间中位数的几倍时，才会被认为该task需要重新启动。</li>
</ul>
<h2 id="3-spark">3. Spark 并行度设置<a class="headerlink" href="#3-spark" title="Permanent link">&para;</a></h2>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1"></a><a href="#__codelineno-2-1"><span class="linenos" data-linenos="1 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.cores.max&quot;</span>, <span class="s2">&quot;4&quot;</span><span class="o">)</span>
<a id="__codelineno-2-2" name="__codelineno-2-2"></a><a href="#__codelineno-2-2"><span class="linenos" data-linenos="2 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.default.parallelism&quot;</span>, <span class="s2">&quot;2&quot;</span><span class="o">)</span>
<a id="__codelineno-2-3" name="__codelineno-2-3"></a><a href="#__codelineno-2-3"><span class="linenos" data-linenos="3 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.streaming.backpressure.enabled&quot;</span>, <span class="s2">&quot;true&quot;</span><span class="o">)</span>
<a id="__codelineno-2-4" name="__codelineno-2-4"></a><a href="#__codelineno-2-4"><span class="linenos" data-linenos="4 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.streaming.receiver.maxRate&quot;</span>, <span class="s2">&quot;1024&quot;</span><span class="o">)</span>
<a id="__codelineno-2-5" name="__codelineno-2-5"></a><a href="#__codelineno-2-5"><span class="linenos" data-linenos="5 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.streaming.kafka.maxRatePerPartition&quot;</span>, <span class="s2">&quot;256&quot;</span><span class="o">)</span>
<a id="__codelineno-2-6" name="__codelineno-2-6"></a><a href="#__codelineno-2-6"><span class="linenos" data-linenos="6 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.streaming.kafka.consumer.poll.ms&quot;</span>, <span class="s2">&quot;4096&quot;</span><span class="o">)</span>
<a id="__codelineno-2-7" name="__codelineno-2-7"></a><a href="#__codelineno-2-7"><span class="linenos" data-linenos="7 "></span></a>.config<span class="o">(</span><span class="s2">&quot;spark.streaming.concurrentJobs&quot;</span>, <span class="s2">&quot;2&quot;</span><span class="o">)</span>
</code></pre></div>
<h2 id="4-spark-on-yarn">4. spark on yarn 调度优化<a class="headerlink" href="#4-spark-on-yarn" title="Permanent link">&para;</a></h2>
<h3 id="41-spark-streamingstreamingcontext-is-started-spark-ui-active-batches-199-sparkexception-could-not-find-coarsegrainedscheduler-yarn">4.1 当并发上来后，经常可能会遇到启动spark streaming程序后会长时间卡在<code>StreamingContext is started</code>上，或者从 spark UI 的大量的 Active Batches (199)，再或者运行一段时间后报错 <code>SparkException: Could not find CoarseGrainedScheduler.</code> 可调整 yarn 参数优化：<a class="headerlink" href="#41-spark-streamingstreamingcontext-is-started-spark-ui-active-batches-199-sparkexception-could-not-find-coarsegrainedscheduler-yarn" title="Permanent link">&para;</a></h3>
<ul>
<li>capacity-scheduler.xml</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1"></a><a href="#__codelineno-3-1"><span class="linenos" data-linenos="1 "></span></a>  <span class="nt">&lt;property&gt;</span>
<a id="__codelineno-3-2" name="__codelineno-3-2"></a><a href="#__codelineno-3-2"><span class="linenos" data-linenos="2 "></span></a>    <span class="nt">&lt;name&gt;</span>yarn.scheduler.capacity.maximum-am-resource-percent<span class="nt">&lt;/name&gt;</span>
<a id="__codelineno-3-3" name="__codelineno-3-3"></a><a href="#__codelineno-3-3"><span class="linenos" data-linenos="3 "></span></a>    <span class="nt">&lt;value&gt;</span>0.8<span class="nt">&lt;/value&gt;</span> <span class="cm">&lt;!-- 默认 0.25，调大可增加 Job 的资源使用率 --&gt;</span>
<a id="__codelineno-3-4" name="__codelineno-3-4"></a><a href="#__codelineno-3-4"><span class="linenos" data-linenos="4 "></span></a>    <span class="nt">&lt;description&gt;</span>Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.<span class="nt">&lt;/description&gt;</span>
</code></pre></div>
<ul>
<li>yarn-site.xml</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1"></a><a href="#__codelineno-4-1"><span class="linenos" data-linenos="1 "></span></a>  <span class="nt">&lt;property&gt;</span>
<a id="__codelineno-4-2" name="__codelineno-4-2"></a><a href="#__codelineno-4-2"><span class="linenos" data-linenos="2 "></span></a>    <span class="nt">&lt;name&gt;</span>yarn.nodemanager.vmem-check-enabled<span class="nt">&lt;/name&gt;</span>
<a id="__codelineno-4-3" name="__codelineno-4-3"></a><a href="#__codelineno-4-3"><span class="linenos" data-linenos="3 "></span></a>    <span class="nt">&lt;value&gt;</span>false<span class="nt">&lt;/value&gt;</span>
<a id="__codelineno-4-4" name="__codelineno-4-4"></a><a href="#__codelineno-4-4"><span class="linenos" data-linenos="4 "></span></a>  <span class="nt">&lt;/property&gt;</span>
<a id="__codelineno-4-5" name="__codelineno-4-5"></a><a href="#__codelineno-4-5"><span class="linenos" data-linenos="5 "></span></a>  <span class="nt">&lt;property&gt;</span>
<a id="__codelineno-4-6" name="__codelineno-4-6"></a><a href="#__codelineno-4-6"><span class="linenos" data-linenos="6 "></span></a>    <span class="nt">&lt;name&gt;</span>yarn.nodemanager.pmem-check-enabled<span class="nt">&lt;/name&gt;</span>
<a id="__codelineno-4-7" name="__codelineno-4-7"></a><a href="#__codelineno-4-7"><span class="linenos" data-linenos="7 "></span></a>    <span class="nt">&lt;value&gt;</span>false<span class="nt">&lt;/value&gt;</span>
<a id="__codelineno-4-8" name="__codelineno-4-8"></a><a href="#__codelineno-4-8"><span class="linenos" data-linenos="8 "></span></a>  <span class="nt">&lt;/property&gt;</span>
</code></pre></div>
<ul>
<li><a href="https://blog.csdn.net/guicaizhou/article/details/83899201">分析: Yarn Capacity Scheduler 配置参数剖析</a></li>
</ul>
<h2 id="5">5. 参考资料<a class="headerlink" href="#5" title="Permanent link">&para;</a></h2>
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/42264669/spark-streaming-assertion-failed-failed-to-get-records-for-spark-executor-a-gro">参考 1: spark streaming assertion failed: Failed to get records for spark-executor-a-group a-topic 7 244723248 after polling for 4096</a></p>
</li>
<li>
<p><a href="https://blog.csdn.net/u013709270/article/details/78879869">参考 2: Could not find CoarseGrainedScheduler</a></p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/SPARK-30745">参考 3: Spark streaming, kafka broker error, "Failed to get records for spark-executor- .... after polling for 512"</a></p>
</li>
<li>
<p><a href="https://issues.apache.org/jira/browse/SPARK-27720">参考 4: 当 kafka rebalancer 后 spark streaming 程序会出现 ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access. 环境: spark 2.3.1 + kafka-1.0.1, see: https://issues.apache.org/jira/browse/SPARK-27720</a></p>
</li>
<li>
<p><a href="https://www.jianshu.com/p/284cb36862ab">参考 5: spark streaming 集成 kafka0.10 offset是如何管理的源码解析</a></p>
</li>
<li>
<p><a href="http://spark.apache.org/docs/2.2.0/streaming-kafka-0-8-integration.html">参考 6: [Official] Spark Streaming + Kafka Integration Guide (Kafka broker version 0.8.2.1 or higher)</a></p>
</li>
<li>
<p><a href="https://blog.csdn.net/jy02268879/article/details/81106417">参考 7: Spark Streaming direct 和 receiver 方式读取的区别</a></p>
</li>
<li>
<p><a href="https://www.cnblogs.com/evakang/p/11043543.html">参考 8: Spark Streaming的优化之路—从 Receiver 到 Direct 模式</a></p>
</li>
<li>
<p><a href="https://help.aliyun.com/document_detail/28124.html">参考 9: 关于 executors 等 https://help.aliyun.com/document_detail/28124.html</a></p>
</li>
<li>
<p><a href="https://www.jianshu.com/p/7ca42da33f8f">参考 10: spark-streaming-kafka010 消费坑 (Failed to get records for spark-executor-abcd1 test1 8 1 after polling for 512)</a></p>
</li>
</ul>

              
            </article>
          </div>
        </div>
        
      </main>
      
        <footer class="md-footer">
  
  <div class="md-footer-meta md-typeset">
    <div class="md-footer-meta__inner md-grid">
      <div class="md-copyright">
  
  
    Made with
    <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
      Material for MkDocs
    </a>
  
</div>
      
    </div>
  </div>
</footer>
      
    </div>
    <div class="md-dialog" data-md-component="dialog">
      <div class="md-dialog__inner md-typeset"></div>
    </div>
    <script id="__config" type="application/json">{"base": "../../../..", "features": ["search.suggest", "search.highlight", "navigation.tabs", "navigation.expand", "toc.follow", "toc.integrate"], "search": "../../../../assets/javascripts/workers/search.5e67fbfe.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version.title": "Select version"}, "version": {"default": "latest", "provider": "mike"}}</script>
    
    
      <script src="../../../../assets/javascripts/bundle.c44cc438.min.js"></script>
      
        <script src="../../../../static/js/util.js"></script>
      
    
  </body>
</html>